• 专利附录
  • 专利说明
  • 权利要求
  • 类似技术
  • 同族专利
  • 引用文献
  • 法律状态
  • 优先权
    • 专利摘要:
      A system and method implements deep learning on a mobile device to provide a convolutional neural network (CNN) for real-time video processing, for example, for coloring hair. The images are processed using CNN to define a respective hair matte of hair pixels. The respective object mattes can be used to determine which pixels to adjust when adjusting the pixel values, such as changing color, lighting, texture, etc. CNN can comprise a (pre-trained) network for image classification adapted to produce the segmentation mask. CNN can be trained for image segmentation (for example, using coarse segmentation data) to minimize a loss of consistency in the mask's image gradient. CNN can also use connection shortcuts between the corresponding layers of an encoder stage and a decoder stage in which the shallower layers of the encoder, which contain high resolution but weak features, are combined with low resolution features, but powerful layers of deeper decoder.
    公开号:BR112020008021A2
    申请号:R112020008021-7
    申请日:2018-10-24
    公开日:2020-10-27
    发明作者:Alex Levinshtein；Cheng Chang；Edmund Phung；Irina Kezele；Wenzhangzhi GUO；Eric ELMOZNINO；Ruowei JIANG；Parham Aarabi
    申请人:L'ORéAL S.A.；
    IPC主号:
    专利说明:

    [001] [001] This application claims, in relation to the United States, the domestic benefit, and in relation to other jurisdictions, of the priority of the Paris Convention for the following applications: 1) United States Provisional Application No. 62 / 576,180 filed in 24 October, 2017 and entitled “A system and method for hair color on video using deep neural networks”; and 2) United States Provisional Order No. 62 / 597,494 filed on December 12, 2017 and entitled “A system and method for matting deep hair in real time on mobile devices”, the entire content of each order is incorporated here by reference to any jurisdiction in which such incorporation is permitted. FIELD OF THE INVENTION
    [002] [002] The present application refers to image processing to define a new image from a source image and, more particularly, to process an image using deep neural networks, such as a convolutional neural network (CNN). BACKGROUND OF THE INVENTION
    [003] [003] There are many scenarios in which image processing is useful for analyzing a source image to identify a certain object-matter and, at least in a subset of these scenarios, for making corrections or other changes to produce a new image. Image processing can be used to classify an object represented in an image and / or to identify the location of the object in an image. Image processing can be used to correct or change attributes (for example, the respective value of a pixel) of an image,
    [004] [004] In one example, image processing can be used to color a subject's hair in a source image to produce a color image of the hair.
    [005] [005] Image processing generally consumes a lot of resources on computing devices, particularly common mobile devices, such as smartphones, tablets etc. This is especially true when processing videos that comprise a series of images in real time. BRIEF DESCRIPTION OF THE INVENTION
    [006] [006] The following description refers to the implementation of deep learning and, in particular, the implementation of deep learning on a mobile device. An objective of the present disclosure is to provide a deep learning environment for processing a live video, for example, to segment an object like hair and change hair color. A person skilled in the art will appreciate that objects other than hair can be detected and color or other attributes can be changed. Video images (for example, frames) can be processed using a deep neural network to define a respective hair matte (for example, an object mask) of hair pixels (for example, object pixels) from each image of video. The respective object mattes can be used to determine which pixels to adjust when adjusting a video image attribute, such as color, lighting, texture, etc. In an example of hair coloring, a deep learning neural network, for example, a convolutional neural network, is configured to classify pixels in a source image to determine whether each is a hair pixel and to define a hair mask. . The mask is then used to change an attribute of the source image to produce a new image. CNN can comprise a pre-trained image classification network adapted to produce the segmentation mask. CNN can still be trained using coarse segmentation data and to minimize a loss of consistency in the mask's image gradient when trained. CNN can also use skip connections between corresponding layers of an encoder phase and a decoder phase where the superficial layers in the encoder, which contain high resolution but weak characteristics are combined with low resolution but strong layer characteristics. deeper into the decoder.
    [007] [007] Such a mask can be used to, directly or indirectly, distinguish other objects. A hair mask can define an individual's margin or border, where the object matter outside the hair mask can be the background, for example. Other objects that can be detected include skin, etc.
    [008] [008] A computing device is provided to process an image comprising: a storage unit for storing and providing a convolutional neural network (CNN) configured to classify image pixels to determine whether each pixel is an object pixel or it is not an object pixel to define an object segmentation mask for an object in the image, in which CNN comprises a pre-trained network for image classification adapted to define the object segmentation mask and in which CNN is still trained using segmentation data; and a processing unit coupled to the storage unit configured to process the image using CNN to generate the object segmentation mask to define a new image.
    [009] [009] CNN can be adapted to minimize a loss of consistency in the mask's image gradient when trained using segmentation data.
    [0010] [0010] CNN can be adapted to use connection shortcuts between layers in an encoder stage and corresponding layers in a decoder stage to combine low resolution but powerful features and high resolution but weak features when doing sampling increase ( upsampling) at the decoder stage to define the object segmentation mask.
    [0011] [0011] The loss of consistency in the mask image gradient can be defined as: 2 ∑
    权利要求:
    Claims (82)
    [1]
    1. COMPUTER DEVICE to process an image, characterized by comprising: a storage unit to store and supply a convolutional neural network (CNN) configured to classify image pixels to determine whether each pixel is an object pixel or not. an object pixel to define an object segmentation mask for an object in the image, in which CNN comprises a pre-trained network for image classification adapted to define the object segmentation mask and in which CNN is further trained using segmentation data; and a processing unit coupled to the storage unit configured to process the image using CNN to generate the object segmentation mask to define a new image.
    [2]
    2. COMPUTER DEVICE, according to claim 1, characterized by CNN being additionally trained to minimize a loss of consistency in the mask image gradient.
    [3]
    3. COMPUTER DEVICE, according to any one of claims 1 to 2, characterized by the loss of consistency in the image gradient of the Lc mask being defined as: 2 ∑
    类似技术:
    公开号 | 公开日 | 专利标题
    BR112020008021A2|2020-10-27|computing devices, method for generating a cnn trained to process images and methods for processing an image
    US10706333B2|2020-07-07|Medical image analysis method, medical image analysis system and storage medium
    US10354362B2|2019-07-16|Methods and software for detecting objects in images using a multiscale fast region-based convolutional neural network
    US20190191988A1|2019-06-27|Screening method for automated detection of vision-degenerative diseases from color fundus images
    US20160071024A1|2016-03-10|Dynamic hybrid models for multimodal analysis
    KR20130071341A|2013-06-28|Face feature vector construction
    CN107977634A|2018-05-01|A kind of expression recognition method, device and equipment for video
    US20200170564A1|2020-06-04|Automatic image-based skin diagnostics using deep learning
    Fuhl et al.2019|500,000 images closer to eyelid and pupil segmentation
    US10891468B2|2021-01-12|Method and apparatus with expression recognition
    Hewitt et al.2018|Cnn-based facial affect analysis on mobile devices
    VARGAS et al.2017|Deep learning: previous and present applications
    WO2022001623A1|2022-01-06|Image processing method and apparatus based on artificial intelligence, and device and storage medium
    Wang et al.2021|Tree leaves detection based on deep learning
    US20210065338A1|2021-03-04|Intelligence-based editing and curating of images
    WO2021057148A1|2021-04-01|Brain tissue layering method and device based on neural network, and computer device
    Elif et al.2020|Accurate Myocardial Pathology Segmentation with Residual U-Net
    KR102152120B1|2020-09-04|Automated Facial Expression Recognizing Systems on N frames, Methods, and Computer-Readable Mediums thereof
    Lin et al.2021|An analysis of English classroom behavior by intelligent image recognition in IoT
    Mauricio et al.2019|A sequential approach for pain recognition based on facial representations
    Sai2021|Image classification for user feedback using Deep Learning Techniques
    US20210224652A1|2021-07-22|Methods and systems for performing tasks on media using attribute specific joint learning
    KR20190130179A|2019-11-22|2d landmark feature synthesis and facial expression strength determination for micro-facial expression detection
    Dahmane et al.2015|Conditional Gabor phase–based disparity estimation applied to facial tracking for person–specific facial action recognition: a preliminary study
    Vazquez Lopez2017|Hand Gesture Recognition for Sign Language Transcription
    同族专利:
    公开号 | 公开日
    US20200320748A1|2020-10-08|
    US11216988B2|2022-01-04|
    WO2019079895A1|2019-05-02|
    CN111448581A|2020-07-24|
    KR20200057770A|2020-05-26|
    EP3701488A4|2021-08-04|
    JP6997309B2|2022-01-17|
    JP2021502627A|2021-01-28|
    RU2743931C1|2021-03-01|
    EP3701488A1|2020-09-02|
    引用文献:
    公开号 | 申请日 | 公开日 | 申请人 | 专利标题
    JP6080417B2|2011-08-19|2017-02-15|キヤノン株式会社|Image processing apparatus and image processing method|
    US9275308B2|2013-05-31|2016-03-01|Google Inc.|Object detection using deep neural networks|
    US9542626B2|2013-09-06|2017-01-10|Toyota Jidosha Kabushiki Kaisha|Augmenting layer-based object detection with deep convolutional neural networks|
    US8965112B1|2013-12-09|2015-02-24|Google Inc.|Sequence transcription with deep neural networks|
    JP2016071868A|2014-09-29|2016-05-09|パナソニックＩｐマネジメント株式会社|Brush area detection device and brush area detection method|
    CN104966093A|2015-05-25|2015-10-07|苏州珂锐铁电气科技有限公司|Dynamic texture identification method based on deep neural networks|
    US9773196B2|2016-01-25|2017-09-26|Adobe Systems Incorporated|Utilizing deep learning for automatic digital image segmentation and stylization|
    US9972092B2|2016-03-31|2018-05-15|Adobe Systems Incorporated|Utilizing deep learning for boundary-aware image segmentation|
    US10867416B2|2017-03-10|2020-12-15|Adobe Inc.|Harmonizing composite images using deep learning|
    US10402689B1|2017-04-04|2019-09-03|Snap Inc.|Generating an image mask using machine learning|JP6333871B2|2016-02-25|2018-05-30|ファナック株式会社|Image processing apparatus for displaying an object detected from an input image|
    US11164067B2|2018-08-29|2021-11-02|Arizona Board Of Regents On Behalf Of Arizona State University|Systems, methods, and apparatuses for implementing a multi-resolution neural network for use with imaging intensive applications including medical imaging|
    DE102018217090A1|2018-10-05|2020-04-09|Robert Bosch Gmbh|Process, artificial neural network, device, computer program and machine-readable storage medium for the semantic segmentation of image data|
    US10866716B2|2019-04-04|2020-12-15|Wheesearch, Inc.|System and method for providing highly personalized information regarding products and services|
    KR20200132340A|2019-05-16|2020-11-25|삼성전자주식회사|Electronic device and Method for controlling the electronic device thereof|
    CN110263731B|2019-06-24|2021-03-16|电子科技大学|Single step human face detection system|
    US10930011B2|2019-07-02|2021-02-23|Billups, Inc.|Digital image processing system for object location and facing|
    US11113526B2|2019-07-23|2021-09-07|Toyota Research Institute, Inc.|Training methods for deep networks|
    TWI725665B|2019-12-13|2021-04-21|宏達國際電子股份有限公司|Image processing system and image processing method|
    CN112580646A|2020-12-08|2021-03-30|北京农业智能装备技术研究中心|Tomato fruit maturity dividing method and picking robot|
    CN112529914B|2020-12-18|2021-08-13|北京中科深智科技有限公司|Real-time hair segmentation method and system|
    法律状态:
    2021-12-07| B350| Update of information on the portal [chapter 15.35 patent gazette]|
    优先权:
    申请号 | 申请日 | 专利标题
    US201762576180P| true| 2017-10-24|2017-10-24|
    US62/576,180|2017-10-24|
    US201762597494P| true| 2017-12-12|2017-12-12|
    US62/597,494|2017-12-12|
    PCT/CA2018/051345|WO2019079895A1|2017-10-24|2018-10-24|System and method for image processing using deep neural networks|
    [返回顶部]